Skip to content

Run all system-tests variants in CI #7538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
smola merged 1 commit into from
Sep 2, 2024
Merged

Run all system-tests variants in CI #7538

smola merged 1 commit into from
Sep 2, 2024

Conversation

@smola
Copy link
Member

@smola smola commented Aug 30, 2024 •
edited by atlassian bot
Loading

What Does This Do

Run all system-tests variants on CI.

Motivation

Maximize functional test coverage.

Additional Notes

Next step will be running all scenarios.

Contributor Checklist

Jira ticket: APPSEC-11137 (partially)

@smola smola added comp: testing Testing tag: no release notes Changes to exclude from release notes labels Aug 30, 2024
@pr-commenter
Copy link

pr-commenter bot commented Aug 30, 2024

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/add-all-system-tests-variants
git_commit_date 1725030291 1725030597
git_commit_sha 7438ef7 4a245d1
release_version 1.39.0-SNAPSHOT~7438ef77c4 1.39.0-SNAPSHOT~4a245d132e
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1725032904 1725032904
ci_job_id 623133619 623133619
ci_pipeline_id 43265023 43265023
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 50 metrics, 13 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.39.0-SNAPSHOT~4a245d132e, baseline=1.39.0-SNAPSHOT~7438ef77c4

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.052 s) : 0, 1051998
Total [baseline] (10.439 s) : 0, 10439004
Agent [candidate] (1.053 s) : 0, 1052565
Total [candidate] (10.388 s) : 0, 10387889
section appsec
Agent [baseline] (1.185 s) : 0, 1184710
Total [baseline] (10.615 s) : 0, 10614922
Agent [candidate] (1.185 s) : 0, 1184703
Total [candidate] (10.606 s) : 0, 10606082
section iast
Agent [baseline] (1.172 s) : 0, 1171975
Total [baseline] (10.859 s) : 0, 10859397
Agent [candidate] (1.181 s) : 0, 1181285
Total [candidate] (10.856 s) : 0, 10856311
section profiling
Agent [baseline] (1.256 s) : 0, 1255555
Total [baseline] (10.634 s) : 0, 10633913
Agent [candidate] (1.257 s) : 0, 1257433
Total [candidate] (10.573 s) : 0, 10572757
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.052 s -
Agent appsec 1.185 s 132.711 ms (12.6%)
Agent iast 1.172 s 119.976 ms (11.4%)
Agent profiling 1.256 s 203.556 ms (19.3%)
Total tracing 10.439 s -
Total appsec 10.615 s 175.918 ms (1.7%)
Total iast 10.859 s 420.393 ms (4.0%)
Total profiling 10.634 s 194.909 ms (1.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.053 s -
Agent appsec 1.185 s 132.138 ms (12.6%)
Agent iast 1.181 s 128.72 ms (12.2%)
Agent profiling 1.257 s 204.867 ms (19.5%)
Total tracing 10.388 s -
Total appsec 10.606 s 218.194 ms (2.1%)
Total iast 10.856 s 468.422 ms (4.5%)
Total profiling 10.573 s 184.868 ms (1.8%) 
gantt
    title petclinic - break down per module: candidate=1.39.0-SNAPSHOT~4a245d132e, baseline=1.39.0-SNAPSHOT~7438ef77c4

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (669.939 ms) : 0, 669939
BytebuddyAgent [candidate] (670.83 ms) : 0, 670830
GlobalTracer [baseline] (307.985 ms) : 0, 307985
GlobalTracer [candidate] (307.255 ms) : 0, 307255
AppSec [baseline] (52.299 ms) : 0, 52299
AppSec [candidate] (52.639 ms) : 0, 52639
Remote Config [baseline] (679.814 µs) : 0, 680
Remote Config [candidate] (687.025 µs) : 0, 687
Telemetry [baseline] (7.514 ms) : 0, 7514
Telemetry [candidate] (7.524 ms) : 0, 7524
section appsec
BytebuddyAgent [baseline] (690.161 ms) : 0, 690161
BytebuddyAgent [candidate] (690.181 ms) : 0, 690181
GlobalTracer [baseline] (300.984 ms) : 0, 300984
GlobalTracer [candidate] (300.834 ms) : 0, 300834
AppSec [baseline] (159.966 ms) : 0, 159966
AppSec [candidate] (159.286 ms) : 0, 159286
IAST [baseline] (21.001 ms) : 0, 21001
IAST [candidate] (21.839 ms) : 0, 21839
Remote Config [baseline] (609.997 µs) : 0, 610
Remote Config [candidate] (615.656 µs) : 0, 616
Telemetry [baseline] (8.889 ms) : 0, 8889
Telemetry [candidate] (9.251 ms) : 0, 9251
section iast
BytebuddyAgent [baseline] (779.338 ms) : 0, 779338
BytebuddyAgent [candidate] (785.334 ms) : 0, 785334
GlobalTracer [baseline] (295.208 ms) : 0, 295208
GlobalTracer [candidate] (297.373 ms) : 0, 297373
AppSec [baseline] (53.417 ms) : 0, 53417
AppSec [candidate] (52.49 ms) : 0, 52490
IAST [baseline] (22.524 ms) : 0, 22524
IAST [candidate] (22.082 ms) : 0, 22082
Remote Config [baseline] (577.805 µs) : 0, 578
Remote Config [candidate] (599.741 µs) : 0, 600
Telemetry [baseline] (7.338 ms) : 0, 7338
Telemetry [candidate] (9.725 ms) : 0, 9725
section profiling
BytebuddyAgent [baseline] (668.672 ms) : 0, 668672
BytebuddyAgent [candidate] (669.132 ms) : 0, 669132
GlobalTracer [baseline] (390.51 ms) : 0, 390510
GlobalTracer [candidate] (391.376 ms) : 0, 391376
AppSec [baseline] (53.181 ms) : 0, 53181
AppSec [candidate] (53.192 ms) : 0, 53192
Remote Config [baseline] (687.857 µs) : 0, 688
Remote Config [candidate] (687.472 µs) : 0, 687
Telemetry [baseline] (7.399 ms) : 0, 7399
Telemetry [candidate] (7.46 ms) : 0, 7460
ProfilingAgent [baseline] (97.018 ms) : 0, 97018
ProfilingAgent [candidate] (97.41 ms) : 0, 97410
Profiling [baseline] (97.042 ms) : 0, 97042
Profiling [candidate] (97.434 ms) : 0, 97434
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.39.0-SNAPSHOT~4a245d132e, baseline=1.39.0-SNAPSHOT~7438ef77c4

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.053 s) : 0, 1053076
Total [baseline] (8.495 s) : 0, 8494977
Agent [candidate] (1.055 s) : 0, 1055412
Total [candidate] (8.526 s) : 0, 8525875
section iast
Agent [baseline] (1.183 s) : 0, 1183443
Total [baseline] (8.984 s) : 0, 8983505
Agent [candidate] (1.177 s) : 0, 1176608
Total [candidate] (8.994 s) : 0, 8994447
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.175 s) : 0, 1174659
Total [baseline] (8.927 s) : 0, 8927400
Agent [candidate] (1.175 s) : 0, 1175023
Total [candidate] (8.935 s) : 0, 8934913
section iast_TELEMETRY_OFF
Agent [baseline] (1.17 s) : 0, 1170300
Total [baseline] (8.964 s) : 0, 8963729
Agent [candidate] (1.173 s) : 0, 1173032
Total [candidate] (8.953 s) : 0, 8953166
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.053 s -
Agent iast 1.183 s 130.366 ms (12.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.175 s 121.582 ms (11.5%)
Agent iast_TELEMETRY_OFF 1.17 s 117.224 ms (11.1%)
Total tracing 8.495 s -
Total iast 8.984 s 488.528 ms (5.8%)
Total iast_HARDCODED_SECRET_DISABLED 8.927 s 432.423 ms (5.1%)
Total iast_TELEMETRY_OFF 8.964 s 468.752 ms (5.5%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.055 s -
Agent iast 1.177 s 121.196 ms (11.5%)
Agent iast_HARDCODED_SECRET_DISABLED 1.175 s 119.611 ms (11.3%)
Agent iast_TELEMETRY_OFF 1.173 s 117.62 ms (11.1%)
Total tracing 8.526 s -
Total iast 8.994 s 468.572 ms (5.5%)
Total iast_HARDCODED_SECRET_DISABLED 8.935 s 409.038 ms (4.8%)
Total iast_TELEMETRY_OFF 8.953 s 427.291 ms (5.0%) 
gantt
    title insecure-bank - break down per module: candidate=1.39.0-SNAPSHOT~4a245d132e, baseline=1.39.0-SNAPSHOT~7438ef77c4

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (671.151 ms) : 0, 671151
BytebuddyAgent [candidate] (673.09 ms) : 0, 673090
GlobalTracer [baseline] (307.466 ms) : 0, 307466
GlobalTracer [candidate] (307.928 ms) : 0, 307928
AppSec [baseline] (52.623 ms) : 0, 52623
AppSec [candidate] (52.517 ms) : 0, 52517
Remote Config [baseline] (672.09 µs) : 0, 672
Remote Config [candidate] (672.798 µs) : 0, 673
Telemetry [baseline] (7.546 ms) : 0, 7546
Telemetry [candidate] (7.558 ms) : 0, 7558
section iast
BytebuddyAgent [baseline] (786.96 ms) : 0, 786960
BytebuddyAgent [candidate] (781.406 ms) : 0, 781406
GlobalTracer [baseline] (297.975 ms) : 0, 297975
GlobalTracer [candidate] (296.848 ms) : 0, 296848
AppSec [baseline] (54.845 ms) : 0, 54845
AppSec [candidate] (53.41 ms) : 0, 53410
IAST [baseline] (21.21 ms) : 0, 21210
IAST [candidate] (21.863 ms) : 0, 21863
Remote Config [baseline] (599.519 µs) : 0, 600
Remote Config [candidate] (585.255 µs) : 0, 585
Telemetry [baseline] (8.121 ms) : 0, 8121
Telemetry [candidate] (8.834 ms) : 0, 8834
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (780.635 ms) : 0, 780635
BytebuddyAgent [candidate] (780.446 ms) : 0, 780446
GlobalTracer [baseline] (296.243 ms) : 0, 296243
GlobalTracer [candidate] (296.246 ms) : 0, 296246
AppSec [baseline] (52.918 ms) : 0, 52918
AppSec [candidate] (52.396 ms) : 0, 52396
IAST [baseline] (22.584 ms) : 0, 22584
IAST [candidate] (23.627 ms) : 0, 23627
Remote Config [baseline] (582.406 µs) : 0, 582
Remote Config [candidate] (584.814 µs) : 0, 585
Telemetry [baseline] (8.066 ms) : 0, 8066
Telemetry [candidate] (8.088 ms) : 0, 8088
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (776.515 ms) : 0, 776515
BytebuddyAgent [candidate] (778.674 ms) : 0, 778674
GlobalTracer [baseline] (296.34 ms) : 0, 296340
GlobalTracer [candidate] (296.704 ms) : 0, 296704
AppSec [baseline] (55.367 ms) : 0, 55367
AppSec [candidate] (53.849 ms) : 0, 53849
IAST [baseline] (20.655 ms) : 0, 20655
IAST [candidate] (21.461 ms) : 0, 21461
Remote Config [baseline] (594.16 µs) : 0, 594
Remote Config [candidate] (584.728 µs) : 0, 585
Telemetry [baseline] (7.199 ms) : 0, 7199
Telemetry [candidate] (8.048 ms) : 0, 8048
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-08-30T15:19:04 2024-08-30T15:25:51
git_branch master smola/add-all-system-tests-variants
git_commit_date 1725030291 1725030597
git_commit_sha 7438ef7 4a245d1
release_version 1.39.0-SNAPSHOT~7438ef77c4 1.39.0-SNAPSHOT~4a245d132e
start_time 2024-08-30T15:18:51 2024-08-30T15:25:38
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1725031893 1725031893
ci_job_id 623133620 623133620
ci_pipeline_id 43265023 43265023
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 18 unstable metrics.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.39.0-SNAPSHOT~4a245d132e, baseline=1.39.0-SNAPSHOT~7438ef77c4
    dateFormat X
    axisFormat %s
section baseline
no_agent (368.448 µs) : 349, 388
.   : milestone, 368,
iast (475.018 µs) : 453, 497
.   : milestone, 475,
iast_FULL (546.428 µs) : 525, 568
.   : milestone, 546,
iast_GLOBAL (496.067 µs) : 475, 518
.   : milestone, 496,
iast_HARDCODED_SECRET_DISABLED (475.26 µs) : 453, 498
.   : milestone, 475,
iast_INACTIVE (448.954 µs) : 428, 470
.   : milestone, 449,
iast_TELEMETRY_OFF (460.902 µs) : 439, 483
.   : milestone, 461,
tracing (431.728 µs) : 411, 452
.   : milestone, 432,
section candidate
no_agent (367.59 µs) : 347, 388
.   : milestone, 368,
iast (476.805 µs) : 454, 499
.   : milestone, 477,
iast_FULL (544.031 µs) : 523, 565
.   : milestone, 544,
iast_GLOBAL (490.061 µs) : 469, 511
.   : milestone, 490,
iast_HARDCODED_SECRET_DISABLED (476.22 µs) : 455, 498
.   : milestone, 476,
iast_INACTIVE (437.673 µs) : 417, 458
.   : milestone, 438,
iast_TELEMETRY_OFF (466.427 µs) : 444, 489
.   : milestone, 466,
tracing (435.884 µs) : 416, 456
.   : milestone, 436,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 368.448 µs [348.552 µs, 388.344 µs] -
iast 475.018 µs [452.721 µs, 497.316 µs] 106.57 µs (28.9%)
iast_FULL 546.428 µs [525.105 µs, 567.752 µs] 177.98 µs (48.3%)
iast_GLOBAL 496.067 µs [474.601 µs, 517.533 µs] 127.619 µs (34.6%)
iast_HARDCODED_SECRET_DISABLED 475.26 µs [452.942 µs, 497.578 µs] 106.811 µs (29.0%)
iast_INACTIVE 448.954 µs [427.99 µs, 469.919 µs] 80.506 µs (21.9%)
iast_TELEMETRY_OFF 460.902 µs [438.99 µs, 482.814 µs] 92.454 µs (25.1%)
tracing 431.728 µs [411.481 µs, 451.975 µs] 63.28 µs (17.2%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 367.59 µs [346.875 µs, 388.304 µs] -
iast 476.805 µs [454.421 µs, 499.189 µs] 109.216 µs (29.7%)
iast_FULL 544.031 µs [522.63 µs, 565.432 µs] 176.441 µs (48.0%)
iast_GLOBAL 490.061 µs [469.052 µs, 511.07 µs] 122.472 µs (33.3%)
iast_HARDCODED_SECRET_DISABLED 476.22 µs [454.637 µs, 497.804 µs] 108.631 µs (29.6%)
iast_INACTIVE 437.673 µs [417.046 µs, 458.301 µs] 70.084 µs (19.1%)
iast_TELEMETRY_OFF 466.427 µs [444.263 µs, 488.591 µs] 98.837 µs (26.9%)
tracing 435.884 µs [415.654 µs, 456.114 µs] 68.295 µs (18.6%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.39.0-SNAPSHOT~4a245d132e, baseline=1.39.0-SNAPSHOT~7438ef77c4
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.342 ms) : 1323, 1361
.   : milestone, 1342,
appsec (1.718 ms) : 1694, 1742
.   : milestone, 1718,
appsec_no_iast (1.719 ms) : 1695, 1742
.   : milestone, 1719,
iast (1.455 ms) : 1433, 1478
.   : milestone, 1455,
profiling (1.51 ms) : 1486, 1534
.   : milestone, 1510,
tracing (1.462 ms) : 1437, 1487
.   : milestone, 1462,
section candidate
no_agent (1.337 ms) : 1318, 1356
.   : milestone, 1337,
appsec (1.705 ms) : 1681, 1730
.   : milestone, 1705,
appsec_no_iast (1.707 ms) : 1683, 1731
.   : milestone, 1707,
iast (1.478 ms) : 1455, 1501
.   : milestone, 1478,
profiling (1.489 ms) : 1465, 1513
.   : milestone, 1489,
tracing (1.451 ms) : 1427, 1476
.   : milestone, 1451,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.342 ms [1.323 ms, 1.361 ms] -
appsec 1.718 ms [1.694 ms, 1.742 ms] 376.107 µs (28.0%)
appsec_no_iast 1.719 ms [1.695 ms, 1.742 ms] 376.816 µs (28.1%)
iast 1.455 ms [1.433 ms, 1.478 ms] 113.531 µs (8.5%)
profiling 1.51 ms [1.486 ms, 1.534 ms] 168.22 µs (12.5%)
tracing 1.462 ms [1.437 ms, 1.487 ms] 120.504 µs (9.0%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.337 ms [1.318 ms, 1.356 ms] -
appsec 1.705 ms [1.681 ms, 1.73 ms] 367.843 µs (27.5%)
appsec_no_iast 1.707 ms [1.683 ms, 1.731 ms] 370.092 µs (27.7%)
iast 1.478 ms [1.455 ms, 1.501 ms] 140.452 µs (10.5%)
profiling 1.489 ms [1.465 ms, 1.513 ms] 151.401 µs (11.3%)
tracing 1.451 ms [1.427 ms, 1.476 ms] 114.155 µs (8.5%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/add-all-system-tests-variants
git_commit_date 1725030291 1725030597
git_commit_sha 7438ef7 4a245d1
release_version 1.39.0-SNAPSHOT~7438ef77c4 1.39.0-SNAPSHOT~4a245d132e
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1725032561 1725032561
ci_job_id 623133621 623133621
ci_pipeline_id 43265023 43265023
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.39.0-SNAPSHOT~4a245d132e, baseline=1.39.0-SNAPSHOT~7438ef77c4
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.453 ms) : 1442, 1465
.   : milestone, 1453,
appsec (2.236 ms) : 2200, 2272
.   : milestone, 2236,
iast (1.972 ms) : 1930, 2015
.   : milestone, 1972,
iast_GLOBAL (2.019 ms) : 1975, 2062
.   : milestone, 2019,
profiling (1.855 ms) : 1820, 1890
.   : milestone, 1855,
tracing (1.831 ms) : 1798, 1863
.   : milestone, 1831,
section candidate
no_agent (1.453 ms) : 1442, 1464
.   : milestone, 1453,
appsec (2.217 ms) : 2181, 2252
.   : milestone, 2217,
iast (1.974 ms) : 1931, 2016
.   : milestone, 1974,
iast_GLOBAL (2.011 ms) : 1967, 2054
.   : milestone, 2011,
profiling (1.857 ms) : 1822, 1892
.   : milestone, 1857,
tracing (1.835 ms) : 1802, 1868
.   : milestone, 1835,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.453 ms [1.442 ms, 1.465 ms] -
appsec 2.236 ms [2.2 ms, 2.272 ms] 782.538 µs (53.8%)
iast 1.972 ms [1.93 ms, 2.015 ms] 519.092 µs (35.7%)
iast_GLOBAL 2.019 ms [1.975 ms, 2.062 ms] 565.182 µs (38.9%)
profiling 1.855 ms [1.82 ms, 1.89 ms] 401.613 µs (27.6%)
tracing 1.831 ms [1.798 ms, 1.863 ms] 377.392 µs (26.0%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.453 ms [1.442 ms, 1.464 ms] -
appsec 2.217 ms [2.181 ms, 2.252 ms] 763.808 µs (52.6%)
iast 1.974 ms [1.931 ms, 2.016 ms] 520.892 µs (35.9%)
iast_GLOBAL 2.011 ms [1.967 ms, 2.054 ms] 557.969 µs (38.4%)
profiling 1.857 ms [1.822 ms, 1.892 ms] 404.385 µs (27.8%)
tracing 1.835 ms [1.802 ms, 1.868 ms] 382.018 µs (26.3%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.39.0-SNAPSHOT~4a245d132e, baseline=1.39.0-SNAPSHOT~7438ef77c4
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.937 s) : 14937000, 14937000
.   : milestone, 14937000,
appsec (15.206 s) : 15206000, 15206000
.   : milestone, 15206000,
iast (18.686 s) : 18686000, 18686000
.   : milestone, 18686000,
iast_GLOBAL (17.538 s) : 17538000, 17538000
.   : milestone, 17538000,
profiling (15.889 s) : 15889000, 15889000
.   : milestone, 15889000,
tracing (15.125 s) : 15125000, 15125000
.   : milestone, 15125000,
section candidate
no_agent (15.089 s) : 15089000, 15089000
.   : milestone, 15089000,
appsec (15.401 s) : 15401000, 15401000
.   : milestone, 15401000,
iast (18.751 s) : 18751000, 18751000
.   : milestone, 18751000,
iast_GLOBAL (18.042 s) : 18042000, 18042000
.   : milestone, 18042000,
profiling (14.95 s) : 14950000, 14950000
.   : milestone, 14950000,
tracing (15.299 s) : 15299000, 15299000
.   : milestone, 15299000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.937 s [14.937 s, 14.937 s] -
appsec 15.206 s [15.206 s, 15.206 s] 269.0 ms (1.8%)
iast 18.686 s [18.686 s, 18.686 s] 3.749 s (25.1%)
iast_GLOBAL 17.538 s [17.538 s, 17.538 s] 2.601 s (17.4%)
profiling 15.889 s [15.889 s, 15.889 s] 952.0 ms (6.4%)
tracing 15.125 s [15.125 s, 15.125 s] 188.0 ms (1.3%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.089 s [15.089 s, 15.089 s] -
appsec 15.401 s [15.401 s, 15.401 s] 312.0 ms (2.1%)
iast 18.751 s [18.751 s, 18.751 s] 3.662 s (24.3%)
iast_GLOBAL 18.042 s [18.042 s, 18.042 s] 2.953 s (19.6%)
profiling 14.95 s [14.95 s, 14.95 s] -139.0 ms (-0.9%)
tracing 15.299 s [15.299 s, 15.299 s] 210.0 ms (1.4%)

@smola smola marked this pull request as ready for review September 2, 2024 06:35
@smola smola requested a review from a team as a code owner September 2, 2024 06:35
@smola smola merged commit dd52c81 into master Sep 2, 2024
@smola smola deleted the smola/add-all-system-tests-variants branch September 2, 2024 07:13
@PerfectSlayer PerfectSlayer added this to the 1.39.0 milestone Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@manuel-alvarez-alvarez manuel-alvarez-alvarez manuel-alvarez-alvarez approved these changes

@ValentinZakharov ValentinZakharov Awaiting requested review from ValentinZakharov ValentinZakharov was automatically assigned from DataDog/asm-java

Assignees

No one assigned

Labels

comp: testing Testing tag: no release notes Changes to exclude from release notes

Projects

None yet

Milestone

1.39.0

Development

Successfully merging this pull request may close these issues.

4 participants

@smola @manuel-alvarez-alvarez @PerfectSlayer